The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
Search systems can be utilized in many different fields. Most search systems utilize crawlers to identify content. A search system can index crawled content, such that the search system is able to search the crawled content. Many applications, however, are difficult or impossible to crawl. For instance, social networking applications such as TWITTER and FACEBOOK have real-time content feeds. In the case of a real-time content feeds, it is difficult for a crawler to crawl all possible updates to all possible content feeds of an application. Further, even if a crawler were to crawl these content feeds, the relevance of the crawled content diminishes as time passes. Thus, at query time, the most relevant content may not yet be indexed. In another example, some applications allow users to provide different types of search parameters when executing specialized searches. For example, the YELP application by Yelp, Inc. allows users to enter different attributes (e.g., cost, cuisine type, location, outdoor seating available, etc) as additional search parameters. This creates a combinatorial explosion of crawlable states of the YELP application due to the vast number of locations and other searchable attributes.